Repeated Commands Based on Device-State Information

ABSTRACT

A portable electronic device with a touch-sensitive display (such as a cellular telephone) provides a wireless remote control for an entertainment device (such as a consumer-electronic device). After receiving, from the portable electronic device, user-interface activity information that specifies a change in a state of the entertainment device, an audio/video (A/V) hub provides the user-interface activity information to the entertainment device and determines device-state information about the entertainment device, which specifies a current state of the entertainment device. Until the device-state information indicates that the change in the state has occurred, the A/V hub: provides: the user-interface activity information to entertainment device; and determines additional device-state information indicating about the entertainment device that indicates whether the change in the state has occurred. In addition, the A/V hub may provide feedback (such as a visual indicator or sensory-feedback instructions) to a user about the status of the change in the state.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims priority under 35 U.S.C. 119(e) to U.S.Provisional Application Ser. No. 62/211,827, “User Interface Based onDevice-State Information,” filed on Aug. 30, 2015, the contents of whichare herein incorporated by reference.

BACKGROUND

Field

The described embodiments relate to feedback techniques, includingdynamically repeating commands until a change in a state of anentertainment device is detected.

Related Art

The versatility and capabilities of portable electronic devices isincreasing their popularity. For example, many portable electronicdevices include touch-sensitive displays that allow users to dynamicallyinteract with the portable electronic devices. In addition, manyportable electronic devices can wirelessly communicate with otherelectronic devices, which allow the portable electronic devices torapidly and conveniently communicate information. In particular, theportable electronic devices may include networking subsystem thatimplement a network interface, such as: a wireless network described inthe Institute of Electrical and Electronics Engineers (IEEE) 802.11standard, Bluetooth® (from the Bluetooth Special Interest Group ofKirkland, Wash.), and/or another type of wireless network. Thecombination of a user interface on a touch-sensitive display andwireless-communication capability can allow users to use portableelectronic devices to remotely control another electronic device.

However, it can be difficult to use user interfaces. For example, manyelectronic devices, which can be wirelessly controlled using portableelectronic devices, have complicated functionality. Because it is oftendifficult to adapt the user interfaces, a user can be forced to navigatethrough a complicated set of options, which may confuse the user.Consequently, the user may make errors, such as activating the wrongfunctionality or getting lost in a large set of options. These errorsfrustrate users, and can degrade the user experience.

SUMMARY

The described embodiments include a portable electronic device. Thisportable electronic device includes: a touch-sensitive display; anantenna; an interface circuit, coupled to the antenna, whichcommunicates with an audio/video (A/V) hub; and a control circuitcoupled to the touch-sensitive display and the interface circuit. Duringoperation, the control circuit receives user-interface activityinformation specifying activation of a virtual command icon in a userinterface displayed on the touch-sensitive display, where the activationof the virtual command icon specifies a change in a state of anentertainment device (such as an A/V display device, e.g., atelevision). Then, the control circuit provides, via the interfacecircuit, the user-interface activity information to the A/V hub.Moreover, the control circuit receives, via the interface circuit,device-state information from the A/V hub indicating whether the changein the state of the entertainment device has occurred. When the changein the state of the entertainment device has not occurred, the controlcircuit performs, until the change in the state of the entertainmentdevice has occurred, operations of: providing, via the interfacecircuit, the user-interface activity information to the A/V hub; andreceiving, via the interface circuit, additional device-stateinformation from the A/V hub indicating whether the change in the stateof the entertainment device has occurred.

Note that, when the change in the state of the entertainment device hasnot occurred, the control circuit may provide, until the change in thestate of the entertainment device has occurred, feedback that indicatesthat the change in the state of the entertainment device is beingimplemented. For example, providing the feedback may involve displayinga visual indicator in the user interface on the touch-sensitive displaythat indicates that the change in the state of the entertainment deviceis being implemented. Alternatively or additionally, providing thefeedback may involve providing, to the A/V hub, instructions for thevisual indicator that indicates that the change in the state of theentertainment device is being implemented, where the visual indicator isto be displayed on the entertainment device.

Furthermore, when the control circuit receives the user-interfaceactivity information again, the control circuit may provide sensoryfeedback that indicates that the change in the state of theentertainment device is being implemented. For example, the portableelectronic device may include a vibration actuator coupled to thecontrol circuit, and providing the sensory feedback may involveactivating the vibration actuator. However, in other embodiments anothertype of sensory feedback is provided.

In some embodiments, the control circuit includes: a processor coupledto the interface circuit; and a memory, coupled to the processor, whichstores a program module that is executed by the processor. The programmodule may include instructions for at least some of the operationsperformed by the portable electronic device.

Another embodiment provides a computer-program product for use with theportable electronic device. This computer-program product includesinstructions for at least some of the operations performed by theportable electronic device.

Another embodiment provides a method for communicating the change in thestate of the entertainment device. This method includes at least some ofthe operations performed by the portable electronic device.

Another embodiment provides the A/V hub, which includes: an antenna; aninterface circuit, coupled to the antenna, which communicates with theportable electronic device (such as a cellular telephone or a remotecontrol); and a control circuit coupled to the interface circuit. Duringoperation, the control circuit in the A/V hub receives, via theinterface circuit, the user-interface activity information from theportable electronic device specifying the activation of the virtualcommand icon in the user interface displayed on the portable electronicdevice, where the activation of the virtual command icon specifies thechange in the state of an entertainment device. Then, the controlcircuit provides the user-interface activity information to theentertainment device. Moreover, the control circuit determines thedevice-state information about the entertainment device indicatingwhether the change in the state of the entertainment device hasoccurred. When the change in the state of the entertainment device hasnot occurred, the control circuit performs, until the change in thestate of the entertainment device has occurred, operations of: providingthe user-interface activity information to the entertainment device; anddetermining additional device-state information about the entertainmentdevice indicating whether the change in the state of the entertainmentdevice has occurred.

Note that, when the change in the state of the entertainment device hasnot occurred, the control circuit may provide, until the change in thestate of the entertainment device has occurred, the feedback thatindicates that the change in the state of the entertainment device isbeing implemented. For example, providing the feedback may involveproviding, via the interface circuit, the visual indicator to theportable electronic device for display in the user interface to indicatethat the change in the state of the entertainment device is beingimplemented. Alternatively or additionally, providing the feedback mayinvolve providing, to the entertainment device, the visual indicator fordisplay on the entertainment device, where the visual indicatorindicates that the change in the state of the entertainment device isbeing implemented.

Furthermore, when the control circuit receives the user-interfaceactivity information again from the portable electronic device, thecontrol circuit may provide, via the interface circuit, asensory-feedback instruction to the portable electronic device that,when performed by the portable electronic device, indicates that thechange in the state of the entertainment device is being implemented.For example, the sensory-feedback instruction may instruct the portableelectronic device to activate the vibration actuator.

In some embodiments, the control circuit includes: a processor coupledto the interface circuit; and a memory, coupled to the processor, whichstores a program module that is executed by the processor. The programmodule may include instructions for at least some of the operationsperformed by the A/V hub.

Another embodiment provides a computer-program product for use with theA/V hub. This computer-program product includes instructions for atleast some of the operations performed by the A/V hub.

Another embodiment provides a method for communicating the change in thestate of the entertainment device. This method includes at least some ofthe operations performed by the A/V hub.

This Summary is provided merely for purposes of illustrating someexemplary embodiments, so as to provide a basic understanding of someaspects of the subject matter described herein. Accordingly, it will beappreciated that the above-described features are merely examples andshould not be construed to narrow the scope or spirit of the subjectmatter described herein in any way. Other features, aspects, andadvantages of the subject matter described herein will become apparentfrom the following Detailed Description, Figures, and Claims.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 is a block diagram illustrating a system with electronic deviceswirelessly communicating in accordance with an embodiment of the presentdisclosure.

FIG. 2 is a flow diagram illustrating a method for identifying anentertainment device in accordance with an embodiment of the presentdisclosure

FIG. 3 is a drawing illustrating communication among the electronicdevices in FIG. 1 in accordance with an embodiment of the presentdisclosure.

FIG. 4 is a flow diagram illustrating a method for displaying a userinterface in accordance with an embodiment of the present disclosure.

FIG. 5 is a drawing illustrating communication among the electronicdevices in FIG. 1 in accordance with an embodiment of the presentdisclosure.

FIG. 6 is a flow diagram illustrating a method for providing a userinterface in accordance with an embodiment of the present disclosure.

FIG. 7 is a drawing illustrating communication among the electronicdevices in FIG. 1 in accordance with an embodiment of the presentdisclosure.

FIG. 8 is a flow diagram illustrating a method for communicating achange in a state of an entertainment device in accordance with anembodiment of the present disclosure.

FIG. 9 is a flow diagram illustrating a method for communicating achange in a state of an entertainment device in accordance with anembodiment of the present disclosure.

FIG. 10 is a drawing illustrating communication among the electronicdevices in FIG. 1 in accordance with an embodiment of the presentdisclosure.

FIG. 11 is a block diagram illustrating a state-detection circuit in oneof the electronic devices of FIG. 1 in accordance with an embodiment ofthe present disclosure.

FIG. 12 is a flow diagram illustrating a method for detecting anentertainment device in accordance with an embodiment of the presentdisclosure.

FIG. 13 is a drawing illustrating a state diagram for one of theelectronic devices in FIG. 1 in accordance with an embodiment of thepresent disclosure.

FIG. 14 is a drawing illustrating a user interface in accordance with anembodiment of the present disclosure.

FIG. 15 is a drawing illustrating a user interface in accordance with anembodiment of the present disclosure.

FIG. 16 is a drawing illustrating a user interface in accordance with anembodiment of the present disclosure.

FIG. 17 is a drawing illustrating a user interface in accordance with anembodiment of the present disclosure.

FIG. 18 is a drawing illustrating a user interface in accordance with anembodiment of the present disclosure.

FIG. 19 is a drawing illustrating a user interface in accordance with anembodiment of the present disclosure.

FIG. 20 is a drawing illustrating a user interface in accordance with anembodiment of the present disclosure.

FIG. 21 is a block diagram illustrating one of the electronic devices ofFIG. 1 in accordance with an embodiment of the present disclosure.

Table 1 provides sets of commands in menus for different entertainmentdevices in accordance with an embodiment of the present disclosure.

Note that like reference numerals refer to corresponding partsthroughout the drawings. Moreover, multiple instances of the same partare designated by a common prefix separated from an instance number by adash.

DETAILED DESCRIPTION

A portable electronic device with a touch-sensitive display (such as acellular telephone) provides a wireless remote control for anentertainment device (such as a consumer-electronic device). Afterreceiving, from the portable electronic device, user-interface activityinformation that specifies a change in a state of the entertainmentdevice, an audio/video (A/V) hub provides the user-interface activityinformation to the entertainment device and determines device-stateinformation about the entertainment device, which specifies a currentstate of the entertainment device. Until the device-state informationindicates that the change in the state has occurred, the A/V hub:provides: the user-interface activity information to entertainmentdevice; and determines additional device-state information indicatingabout the entertainment device that indicates whether the change in thestate has occurred. In addition, the A/V hub may provide feedback (suchas a visual indicator or sensory-feedback instructions) to a user aboutthe status of the change in the state.

By providing feedback to the user about the status of the change in thestate of the entertainment device, this feedback technique may provideintuitive information to the user. For example, the feedback may providesituation awareness to the user, so that the user does not repeatedlyattempt to activate the virtual command icon. In addition, the feedbackmay help keep the user from becoming frustrated (which may occur inexisting approaches where the user is unaware about the status of thechange in the state of the entertainment device). Therefore, thefeedback technique may make it easier for the user to use the portableelectronic device as a wireless remote control, and may allow the userto effectively use the user interface to control the entertainmentdevice (such as the A/V hub, an A/V display device and/or aconsumer-electronic device) with fewer errors or mistakes. Consequently,the feedback technique may improve the user experience when using theportable electronic device, the A/V hub and the entertainment device.

In the discussion that follows the portable electronic device, the A/Vhub and/or the A/V display device may include radios that communicatepackets or frames in accordance with one or more communicationprotocols, such as: an Institute of Electrical and Electronics Engineers(IEEE) 802.11 standard (which is sometimes referred to as ‘Wi-Fi®,’ fromthe Wi-Fi® Alliance of Austin, Tex.), Bluetooth® (from the BluetoothSpecial Interest Group of Kirkland, Wash.), a cellular-telephonecommunication protocol, a near-field-communication standard orspecification (from the NFC Forum of Wakefield, Mass.), and/or anothertype of wireless interface. In the discussion that follows, Wi-Fi isused as an illustrative example. For example, the cellular-telephonecommunication protocol may include or may be compatible with: a 2^(nd)generation of mobile telecommunication technology, a 3^(rd) generationof mobile telecommunications technology (such as a communicationprotocol that complies with the International MobileTelecommunications-2000 specifications by the InternationalTelecommunication Union of Geneva, Switzerland), a 4^(th) generation ofmobile telecommunications technology (such as a communication protocolthat complies with the International Mobile Telecommunications Advancedspecification by the International Telecommunication Union of Geneva,Switzerland), and/or another cellular-telephone communication technique.In some embodiments, the communication protocol includes Long TermEvolution or LTE. However, a wide variety of communication protocols maybe used. In addition, the communication may occur via a wide variety offrequency bands. Note that the portable electronic device, the A/V huband/or the A/V display device may communicate using infra-redcommunication that is compatible with an infra-red communicationstandard (including unidirectional or bidirectional infra-redcommunication).

Communication among electronic devices is shown in FIG. 1, whichpresents a block diagram illustrating a system 100 with a portableelectronic device 110 (such as a remote control or a cellulartelephone), an A/V hub 112, A/V display device 114 (such as atelevision, a monitor, a computer and, more generally, a displayassociated with an electronic device) and one or moreconsumer-electronic devices 116 (e.g., a radio receiver, a video player,a satellite receiver, an access point that provides a connection to awired network such as the Internet, a media or a content source, aconsumer-electronic device, a set-top box, over-the-top contentdelivered over the Internet or a network without involvement of a cable,satellite or multiple-system operator, etc.). (Note that A/V hub 112,A/V display device 114, and the one or more consumer-electronic devices116 are sometimes collectively referred to as ‘components’ in system100. However, A/V hub 112, A/V display device 114, and the one or moreconsumer-electronic devices 116 are sometimes referred to asentertainment devices.) In particular, portable electronic device 110and A/V hub 112 may communicate with each other using wirelesscommunication, and A/V hub 112 and other components in system 100 (suchas A/V display device 114 and the one or more consumer-electronicdevices 116) may communicate using wireless and/or wired communication.During the wireless communication, these electronic devices maywirelessly communicate while: transmitting advertising frames onwireless channels, detecting one another by scanning wireless channels,establishing connections (for example, by transmitting associationrequests), and/or transmitting and receiving packets or frames (whichmay include the association requests and/or additional information aspayloads, such as user-interface information, device-state information,user-interface activity information, data, A/V content, etc.).

As described further below with reference to FIG. 21, portableelectronic device 110, A/V hub 112, A/V display device 114 and the oneor more consumer-electronic devices 116 may include subsystems, such as:a networking subsystem, a memory subsystem and a processor subsystem. Inaddition, portable electronic device 110 and A/V hub 112, and optionallyone or more of A/V display device 114 and/or the one or moreconsumer-electronic devices 116, may include radios 118 in thenetworking subsystems. (Note that radios 118 may be instances of thesame radio or may be different from each other.) More generally,portable electronic device 110 and A/V hub 112 (and optionally one ormore of A/V display device 114 and/or the one or moreconsumer-electronic devices 116) can include (or can be included within)any electronic devices with the networking subsystems that enableportable electronic device 110 and A/V hub 112 (and optionally one ormore of A/V display device 114 and/or the one or moreconsumer-electronic devices 116) to wirelessly communicate with eachother. This wireless communication can comprise transmittingadvertisements on wireless channels to enable electronic devices to makeinitial contact or detect each other, followed by exchanging subsequentdata/management frames (such as association requests and responses) toestablish a connection, configure security options (e.g., InternetProtocol Security), transmit and receive packets or frames via theconnection, etc.

As can be seen in FIG. 1, wireless signals 120 (represented by a jaggedline) are transmitted from radio 118-1 in portable electronic device110. These wireless signals are received by at least A/V hub 112. Inparticular, portable electronic device 110 may transmit packets. Inturn, these packets may be received by a radio 118-2 in A/V hub 112.This may allow portable electronic device 110 to communicate informationto A/V hub 112. While FIG. 1 illustrates portable electronic device 110transmitting packets, note that portable electronic device 110 may alsoreceive packets from A/V hub 112.

In the described embodiments, processing of a packet or frame inportable electronic device 110 and A/V hub 112 (and optionally one ormore of A/V display device 114 and/or the one or moreconsumer-electronic devices 116) includes: receiving wireless signals120 with the packet or frame; decoding/extracting the packet or framefrom received wireless signals 120 to acquire the packet or frame; andprocessing the packet or frame to determine information contained in thepacket or frame (such as the information associated with a data stream).For example, the information from portable electronic device 110 mayinclude user-interface activity information associated with a userinterface displayed on touch-sensitive display 124 in portableelectronic device 110, which a user of portable electronic device 110uses to control A/V hub 112, A/V display device 114 and/or one of theone or more consumer-electronic devices 116. Alternatively, theinformation from A/V hub 112 may include device-state information abouta current device state of A/V display device 114 or one of the one ormore consumer-electronic devices 116 (such as on, off, play, rewind,fast forward, a selected channel, selected content, a content source,etc.), or may include user-interface information for the user interface(which may be dynamically updated based on the device-state informationand/or the user-interface activity information). Furthermore, theinformation from A/V hub 112 and/or one of the one or moreconsumer-electronic devices 116 may include audio and video that aredisplayed on A/V display device 114. (However, as noted previously, theaudio and video may be communicated between components in system 100 viawired communication. Therefore, as shown in FIG. 1, there may be a wiredcable or link, such as a high-definition multimedia-interface (HDMI)cable 122, between A/V hub 112 and A/V display device 114.)

Note that the communication between portable electronic device 110 andA/V hub 112 (and optionally one or more of A/V display device 114 and/orthe one or more consumer-electronic devices 116) may be characterized bya variety of performance metrics, such as: a data rate, a data rate forsuccessful communication (which is sometimes referred to as a‘throughput’), an error rate (such as a retry or resend rate), amean-square error of equalized signals relative to an equalizationtarget, intersymbol interference, multipath interference, asignal-to-noise ratio, a width of an eye pattern, a ratio of number ofbytes successfully communicated during a time interval (such as 1-10 s)to an estimated maximum number of bytes that can be communicated in thetime interval (the latter of which is sometimes referred to as the‘capacity’ of a channel or link), and/or a ratio of an actual data rateto an estimated data rate (which is sometimes referred to as‘utilization’). Moreover, the performance during the communicationassociated with different channels may be monitored individually orjointly (e.g., to identify dropped packets).

The communication between portable electronic device 110 and A/V hub 112(and optionally one or more of A/V display device 114 and/or the one ormore consumer-electronic devices 116) in FIG. 1 may involve one or moreindependent, concurrent data streams in different wireless channels (oreven different Wi-Fi communication protocols) in one or more connectionsor links, which may be communicated using multiple radios. Note that theone or more connections or links may each have a separate or differentservice set identifier on a wireless network in system 100 (which may bea proprietary network or a public network). Moreover, the one or moreconcurrent data streams may, on a dynamic or packet-by-packet basis, bepartially or completely redundant to improve or maintain the performancemetrics even when there are transient changes (such as interference,changes in the amount of information that needs to be communicated,movement of portable electronic device 110, etc.), and to facilitateservices (while remaining compatible with the Wi-Fi communicationprotocol) such as: channel calibration, determining of one or moreperformance metrics, performing quality-of-service characterizationwithout disrupting the communication (such as performing channelestimation, determining link quality, performing channel calibrationand/or performing spectral analysis associated with at least onechannel), seamless handoff between different wireless channels,coordinated communication between components, etc. These features mayreduce the number of packets that are resent, and, thus, may decreasethe latency and avoid disruption of the communication and may enhancethe experience of one or more users or viewers of content on A/V displaydevice 114.

As noted previously, a user may control A/V hub 112, A/V display device114 and/or one of the one or more consumer-electronic devices 116 viathe user interface displayed on touch-sensitive display 124 on portableelectronic device. In particular, at a given time, the user interfacemay include one or more virtual icons that allow the user to activate,deactivate or change functionality or capabilities of A/V hub 112, A/Vdisplay device 114 and/or one of or more consumer-electronic devices116. For example, a given virtual icon in the user interface may have anassociated strike area on a surface of touch-sensitive display 124. Ifthe user makes and then breaks contact with the surface (e.g., using oneor more fingers or digits, or using a stylus) within the strike area,portable electronic device 110 (such as a processor executing a programmodule) may receive user-interface activity information indicatingactivation of this command or instruction from a touch-screeninput/output (I/O) controller, which is coupled to touch-sensitivedisplay 124. (Alternatively, touch-sensitive display 124 may beresponsive to pressure. In these embodiments, the user may maintaincontact with touch-sensitive display 124 with an average contactpressure that is usually less than a threshold value, such as 10-20 kPa,and may activate a given virtual icon by increase the average contactpressure with touch-sensitive display 124 above the threshold value.) Inresponse, the program module may instruct an interface circuit inportable electronic device 110 to wirelessly communicate theuser-interface activity information indicating the command orinstruction to A/V hub 112, and A/V hub 112 may communicate the commandor the instruction to the target component in system 100 (such as A/Vdisplay device 114). This instruction or command may result in A/Vdisplay device 114 turning on or off, displaying content from aparticular source, performing a trick mode of operation (such as fastforward, reverse, fast reverse or skip), etc.

One problem with using existing remote controls to control the operationof another component or entertainment device is that the remote controldoes not receive any feedback from the entertainment device. Forexample, many existing remote controls use infra-red communication.However, typically existing infra-red communication protocols areunidirectional or one-way communication, i.e., from a remote control tothe entertainment device. Consequently, the remote control usually doesnot have any knowledge of the effects of the commands or instructionsthat are communicated to the entertainment device. In particular, theremote control is typically unaware of a current state of theentertainment device, such as whether the entertainment device is in: apower-on state, a power-off state, a playback state, a trick-mode state(such as fast forward, fast reverse, or skip), a pause state, a standby(reduced-power) state, a record state, a state in which contentassociated with a given content source (such as cable television, asatellite network, a web page on the Internet, etc.) is received orprovided, and/or another state. (Note that one or more of the states maybe nested or concurrent with each other, such as the power-on state andthe playback state.) By operating blindly in this way, existing remotecontrol are unable to leverage knowledge of the current state of theentertainment device to improve the user experience.

This problem is addressed in system 100. In particular, as describedfurther below with reference to FIGS. 2 and 3, A/V hub 112 may determinethe current state of one or more of the components in system 100, suchas the current state of A/V display device 114 and/or one of the one ormore consumer-electronic devices 116. This device-state information maybe determined by A/V hub 112 using hardware and/or software, and A/V hub112 may determine the device-state information even for legacyentertainment devices that are only capable of receiving commands orinstructions (i.e., that are only capable of unidirectionalcommunication). For example, as described further below with referenceto FIGS. 11 and 12, whether or not a given component or entertainmentdevice in system 100 is electrically coupled to A/V hub 112 may bedetermined using a state-detection circuit that detects whether there iselectrical coupling between the entertainment device and an inputconnector to A/V hub 112 (such as an HDMI connector or port that can beelectrically coupled to HDMI cable 122). If the electrical coupling isdetected, the type of the given entertainment device (such as atelevision, a DVD player, a satellite receiver, etc.) and/or themanufacturer or provider of the given entertainment device may bedetermined by A/V hub 112 by providing a series of commands orinstructions to the given entertainment device (e.g., such as commandsor instructions that are specific to a particular type of entertainmentdevice, specific to a particular manufacturer, and/orconsumer-electronics-control commands in the HDMI standard orspecification), and then monitoring, as a function of time, changes in adata stream (as indicated by the number of packets or frames and/or thepayloads in the packets or frames) to and/or from the givenentertainment device to see if there was a response to a particularcommand or instruction. Moreover, the state-detection circuit maydetermine whether the given entertainment device is in the power-onstate or the power-off state by monitoring a voltage, a current and/oran impedance on, through or associated with one or more pins in theinput connector. Alternatively or additionally, A/V hub 112 maydetermine whether the given entertainment device is in the power-onstate or the power-off state by monitoring, as a function of time, thedata stream (as indicated by the number of packets or frames and/or thepayloads in the packets or frames) to and/or from the givenentertainment device. Similarly, A/V hub 112 may determine the currentstate of the given entertainment device, such as whether the givenentertainment device responded to a command or instruction that wasprovided to the given entertainment device by A/V hub 112, bymonitoring, as a function of time, changes in the data stream (asindicated by the number of packets or frames and/or the payloads in thepackets or frames) to and/or from the given entertainment device. Thus,the device-state information for the given entertainment devicedetermined by A/V hub 112 may include: presence or absence information(such as whether there is electrical coupling or a wireless connectionwith the given entertainment device), identity information (such as thetype of the given entertainment device and/or the manufacturer of thegiven entertainment device) and/or the current state.

Using the device-state information A/V hub 112 and/or portableelectronic device 110 may dynamically adapt the user interface displayedon touch-sensitive display 124 on portable electronic device 110. Forexample, as described further below with reference to FIGS. 4 and 5, A/Vhub 112 may provide, via radio 118-2, device-state information toportable electronic device 110 specifying a current state of the givenentertainment device. (Thus, the feedback technique may includebidirectional or two-way communication between A/V hub 112 and portableelectronic device 110.) As shown in FIGS. 14-20, after radio 118-1receives the device-state information, portable electronic device 110(such as a program module executed in an environment, e.g., an operatingsystem, in portable electronic device 110) may generate a user interfacethat includes one or more virtual command icons associated with thecurrent state and one or more related states of the given entertainmentdevice. As illustrated by the available commands shown in FIG. 13, theone or more related states may be related to the current state in astate diagram (which may be stored in memory in portable electronicdevice 110) by corresponding operations that transition the givenentertainment device from the current state to the one or more relatedstates. Then, portable electronic device 110 may display the userinterface on touch-sensitive display 124.

In some embodiments, A/V hub 112 provides information specifying thetype of the given entertainment device, the manufacturer of the givenentertainment device, and/or context information that specifies acontext of content (such as A/V content) displayed on the entertainmentdevice (such as A/V display device 114). For example, the context mayinclude a type of the content (such as sports, television, a movie,etc.), a location in the content (such as a timestamp, an identifier ofa sub-section in the content and/or a proximity to a beginning or an endof the content), etc. In these embodiments, the one or more virtualcommand icons (and, thus, the user interface) may be based on the typeof the given entertainment device, the manufacturer and/or the context.Thus, only virtual command icons that are relevant to the givenentertainment device, the manufacturer and/or the context may beincluded in the user interface.

Moreover, when the user activates one of the virtual command icons inthe user interface, the touch-screen I/O controller in portableelectronic device 110 may provide user-interface activity informationspecifying activation of a virtual command icon in the one or morevirtual command icons, where the activation of the virtual command iconspecifies a transition of the given entertainment device from thecurrent state to a new current state in the state diagram. As notedpreviously, the activation of the virtual command icon may involve auser of portable electronic device 110 contacting touch-sensitivedisplay 124 within a strike area of the virtual command icon and thenreleasing the contact. In response to receiving the user-interfaceactivity information, portable electronic device 110 may: modify theuser interface to change the one or more virtual command icons based onthe new current state; and display the modified user interface ontouch-sensitive display 124. Note that portable electronic device 110may wait to change the one or more virtual command icons until thedevice-state information received from A/V hub 112 indicates that thegiven entertainment device has transitioned to the new current state inresponse to a command or an instruction associated with the activationof the one of the virtual command icons. Thus, portable electronicdevice 110 may repeatedly perform the generating and the displayingoperations so that the user interface is dynamically updated as thecurrent state changes.

Alternatively or additionally, as described further below with referenceto FIGS. 6 and 7, instead of portable electronic device 110 generatingthe user interface, A/V hub 112 may generate user-interface informationthat specifies the user interface (or instructions specifying objects orgraphical information in the user interface) based on the one or morerelated states in the state diagram (which may be stored in memory inA/V hub 112) and one or more of: the device-state information, the typeof the given entertainment device, the manufacturer of the givenentertainment device, the context, and/or user-interface activityinformation specifying activation (by the user) of one of the virtualcommand icons in the user interface (which may be received, via radios118, from portable electronic device 110). Then, A/V hub 112 mayprovide, via radios 118, the user-interface information to portableelectronic device 110 for display on touch-sensitive display 124.

Furthermore, as described further below with reference to FIGS. 8-10,the device-state information may be used by portable electronic device110 to ensure that commands or instructions have been received andexecuted by the designated recipient entertainment device in system 100.If the device-state information indicates that a command or instructionwas not executed, portable electronic device 110 may resent the commandor instruction. Alternatively, if the device-state information indicatesthat the command or instruction is being processed, portable electronicdevice 110 may not resent the command or instruction, and may providefeedback to the user of portable electronic device 110 so that the useris aware that the command or instruction is being processed and may notcontinue to activate a virtual command icon in the user interface infrustration.

The feedback may originate on portable electronic device 110 and/or A/Vhub 112. For example, in response to device-state information, portableelectronic device 110 (such as a processor executing a program module)may: generate and display in the user interface a visual indicator thatthe change in the state of the entertainment device is beingimplemented; and/or activate a sensory-feedback mechanism (such as avibration mechanism, a vibration actuator, a light, or a speaker).Alternatively, A/V hub 112 (such as a processor executing a programmodule) may: generate and provide, via the interface circuit in A/V hub112, the visual indicator to portable electronic device 110 for displayin the user interface to indicate that the change in the state of theentertainment device is being implemented; and/or provide asensory-feedback instruction that activates the sensory-feedbackmechanism in portable electronic device 110.

In some embodiments, A/V hub 112 may generate the visual indicator basedon the user-interface activity information. Then, A/V hub 112 mayprovide, via the interface circuit in A/V hub 112, the visual indicatorto A/V display device 114 for display on A/V display device 114. Notethat the visual indicator displayed on A/V display device 114 mayindicate that the function associated with the virtual command icon isbeing processed or implemented. For example, the visual indicator mayinclude graphical information, such as flashing a representation of thevirtual command icon, changing a line thickness in the virtual commandicon and/or adding a graphical symbol (such as an hour glass or a watchface). Note that A/V display device 114 may display the visual indicatoron the A/V display device along with content, such as A/V content thatis generated by A/V hub 112 and/or one of the one or moreconsumer-electronic devices 116. In particular, the visual indicator maybe superimposed on or over the A/V content. Moreover, the visualindicator may be partially transparent so that the A/V content isvisible underneath the visual indicator when displayed on A/V displaydevice 114. In an exemplary embodiment, the visual indicator may includespatial information or graphical information that summarizes the currentspatial configuration of the user interface (including one or morevirtual icons, their functions and/or the associated strike areas), aswell as the currently activated virtual command icon.

Furthermore, the visual indicator may include or specify a currentcontact area of a user's finger(s), a stylus or a pen on touch-sensitivedisplay 124, which may allow the user to navigate to a virtual commandicon by looking at A/V display device 114 instead of the user interfacedisplayed on touch-sensitive display 124. This capability may be usefulto the user, such as in cold environments where the user and portableelectronic device 110 may be covered by a blanket. In a variation onthis embodiment, the visual indicator may specify or represent a currentlocation in the user interface and/or touch-sensitive display 124 thatis activated. This may be useful in embodiments where the user interactswith the user interface without contacting the surface oftouch-sensitive display 124 (such as based on time-of-flightmeasurements, a laser pointer, etc.).

In this way, the user interface may be dynamically updated as thecomponents in system 100 respond to commands or instructions receivedfrom portable electronic device 110 and/or A/V hub 112, so that thecurrently relevant one or more virtual icons are included in the userinterface. This capability may simplify the user interface and make iteasier for the user to navigate through and/or use the user interface.Alternatively or additionally, portable electronic device 110 may ensurethat commands or instructions are resent, as needed. Consequently, thefeedback technique may reduce user errors and/or confusion when usingportable electronic device 110 and/or A/V hub 112, and may ensure thatcomponents in system 100 respond to user commands and instructions,which individually and/or collectively may improve the user experience.

Although we describe the network environment shown in FIG. 1 as anexample, in alternative embodiments, different numbers or types ofelectronic devices may be present. For example, some embodimentscomprise more or fewer electronic devices. As another example, inanother embodiment, different electronic devices are transmitting and/orreceiving packets or frames. While portable electronic device 110 andA/V hub 112 are illustrated with a single instance of radios 118, inother embodiments portable electronic device 110 and A/V hub 112 (andoptionally one or more of A/V display device 114 and/or the one or moreconsumer-electronic devices 116) may include multiple radios.

We now describe embodiments of a feedback technique. FIG. 2 presents aflow diagram illustrating method 200 for identifying an entertainmentdevice, which may be performed by an A/V hub, such as A/V hub 112 (FIG.1). This A/V hub may include: an input connector that can electricallycouple to an entertainment device, where the input connector iscompatible with a high-definition multimedia-interface (HDMI) standard;and a state-detection circuit coupled to at least one pin in the inputconnector (such as a transition minimized differential signaling or TMDSdata1 shield), where, when the entertainment device is electricallycoupled to the input connector, the state-detection circuit establishesa ground loop between the electronic device and the entertainmentdevice. During operation, the A/V hub (such as a control circuit orcontrol logic, e.g., a processor executing a program module and/or or acircuit, which is electrically coupled to the input connector and/or thestate-detection circuit) detects whether there is electrical coupling(operation 210) between the entertainment device and the input connectorusing the state-detection circuit.

When the electrical coupling is detected (operation 210), the A/V hubprovides a set of first control commands (operation 216) associated withdifferent types of entertainment devices until, in response, the A/V hubdetects content activity (operation 218) via the input connector. Notethat the set of first control commands may include: power-on controlcommands for the different providers of entertainment devices; and/orpower-off control commands for the different providers of entertainmentdevices. Moreover, the set of first commands may include: a play commandfor the different types of entertainment devices; and/or a trick-modecommand for the different types of entertainment devices. (Moregenerally, the set of first control commands may include commandsassociated with a variety of states of the entertainment device and/orwhich may result in a variety of changes in the state of theentertainment device.) Furthermore, the content activity may include A/Vcontent that is at least one of provided and received by theentertainment device. In an exemplary embodiment, power-on controlcommands or play commands for different types of entertainment devices(such as a DVD player, a satellite receiver, etc.) may be provided tothe entertainment device until the content activity is detected (such assignificant increase or decrease in the number of packets or framescommunicated to and/or from the entertainment device).

When the content activity is detected (operation 218), the A/V hubprovides a set of second control commands (operation 220) associatedwith different providers of entertainment devices until the A/V hubdetects a change in a state (operation 222) of the entertainment devicevia the input connector and the state-detection circuit. Note that theset of second control commands may include commands associated with avariety of states of the entertainment device and/or which may result ina variety of changes in the state of the entertainment device. In anexemplary embodiment, power-off control commands or standby controlcommands associated with different providers for the type ofentertainment device may be provided to the entertainment device untilthe change in the state is detected (such as significant change in thenumber of packets or frames communicated to and/or from theentertainment device).

Furthermore, when the electrical coupling between the entertainmentdevice and the input connector is detected (operation 210) and beforeproviding the set of first control commands (operation 216), the A/V hubmay attempt to identify the entertainment device by optionally providingconsumer-electronics-control commands (operation 212) to theentertainment device (which may include commands supported by theConsumer Electronics Control capability supported by the HDMI standardand certain entertainment devices). If the attempt is unsuccessful(operation 214), then the A/V hub may provide the set of first controlcommands (operation 216). (Thus, the A/V hub may use the set of firstcontrol commands and/or a set of second control commands to identify atleast some entertainment devices without using the Consumer ElectronicsControl capability.)

As described further below with reference to FIG. 11, thestate-detection circuit may include: an energy-dissipation component(such as a resistor) electrically coupled to a power-supply voltage andat least the one pin; an energy-storage component (such as a capacitor)electrically coupled to at least the one pin and ground; and abi-directional voltage clamp (such as a varistor or a Verner diode), inparallel with the energy-storage component, electrically coupled to atleast the one pin and ground.

Moreover, detecting whether there is electrical coupling between theentertainment device and the input connector (operation 210) mayinvolve: setting at least the one pin as an input, where at least theone pin is then pulled to the power-supply voltage by the A/V hub;measuring a voltage on at least the one pin; and detecting theelectrical coupling between the entertainment device and the inputconnector when the voltage on at least the one pin is less than or equalto a predefined value (such as when the voltage on at least the one pinis approximately ground).

Furthermore, when the electrical coupling between the entertainmentdevice and the input connector is detected (operation 210), the A/V hubmay: set at least the one pin as an output and electrically couple atleast the one pin to ground; and measure a second voltage on ahotplug-detect pin in the input connector. When the second voltage onthe hotplug-detect pin is less than or equal to the predefined value,the A/V hub may set at least the one pin as an input and repeat themeasurement of the voltage on at least the one pin. Alternatively, whenthe voltage equals or exceeds a second predefined value (such as whenthe voltage is approximately the power-supply voltage), the A/V hub mayrepeat the detecting whether there is electrical coupling between theentertainment device and the input connector (operation 210).Additionally, when the voltage is less than or equal to the predefinedvalue, the A/V hub may identify the state of the entertainment device(e.g., the A/V hub may provide the set of first control commands,provide the set of second control commands, and/or may monitor thecontent activity, such as a data stream to and/or from the entertainmentdevice). Note that the state may include: powered off; and standby.However, the state may include a variety of other states.

Additionally, when the second voltage on the hotplug-detect pin is lessthan or equal to the predefined value and when the voltage is less thanor equal to the predefined value, the A/V hub may repeat setting atleast the one pin as the output and electrically coupling at least theone pin to ground.

Note that based on the first control command(s) that resulted in thecontent activity and the second control command(s) that resulted in thechange in the state, the entertainment device may be identified.Moreover, based on the identified entertainment device, the A/V hub mayaccess one or more predefined and/or predetermined commands associatedwith the entertainment device. Then, using device-state information forthe entertainment device, a user interface displayed on the portableelectronic device and/or on a display in an A/V hub display device maybe dynamically adapted or adjusted.

FIG. 3 presents a drawing illustrating communication among theelectronic devices in FIG. 1, which presents a drawing illustratingcommunication between portable electronic device 110, A/V hub 112 andentertainment device 310. In particular, processor 310 andstate-detection circuit 312 may detect electrical coupling 314 withentertainment device 318. When electrical coupling 314 is detected,processor 310 may instruct interface circuit 316 to provide the set offirst control commands 320 to entertainment device 318. Then, processor310 may, via interface circuit 316, monitor content activity 322associated with entertainment device 318 (such as content activityrepresented by a data stream between entertainment device 318 and A/Vhub 112). Alternatively or additionally, processor 310 may, viastate-detection circuit 312, determine a change in power state 324.

When content activity 322 and/or a change in power state 324 is detected(which indicates that entertainment device 318 responded to a particularfirst control command, and thus specifies a type of entertainment device318), processor 310 may instruct interface circuit 316 to provide theset of second control commands 326 to entertainment device 318.Furthermore, processor 310 may, via interface circuit 316, monitorcontent activity 328 associated with entertainment device 310 (such ascontent activity represented by a data stream between entertainmentdevice 318 and A/V hub 112). Alternatively or additionally, processor310 may, via state-detection circuit 312, determine a change in powerstate 330. A change in content activity 328 and/or power state 330 for aparticular second control command may specify the provider ofentertainment device 318.

Using the type of the entertainment device and the provider ofentertainment device 318, processor 310 may identify 332 entertainmentdevice 318.

In an exemplary embodiment, when state-detection circuit 312 detectselectrical coupling 314 with entertainment device 318, processor 310instructs interface circuit 316 to provide power-on commands for certainmanufacturers or providers of consumer-electronic devices (orentertainment devices) and/or for certain types or classes ofconsumer-electronic devices. A response to a particular power-on commandmay provide initial classification information, which may specify themanufacturer or provider of a consumer-electronic device, or a class ortype of consumer-electronic device (such as a DVD player). Note that thepower-on commands may be provided in an intelligent manner. For example,the power-on command for a low-probability consumer-electronic device(such as a regular, non-Blu-ray DVD player) may be provided last. Incontrast, the power-on command for a high-probabilityconsumer-electronic device (such as a set-top box), which may be themajority of the consumer-electronic devices that are used with the A/Vhub, may be provided first.

After determining the initial classification information, processor 310may instruct interface circuit 316 to provide play or another menucommand(s) to refine the classification information by determining asub-type classification. For example, there may be three different typesof DVD players from a particular manufacturer (which are used as anillustration of a consumer-electronic device). Based on differences inthe menu of commands available to a remote control for the threedifferent types of DVD players, the other menu command(s) may beselected. The response to the other menu command(s) may providedifferential information that allows the classification information tobe refined. Thus, a play command may be sent to one type of DVD player,and a stop command may be sent to another type of DVD player.

The determined classification information may allow the user interfaceto be automatically customized for the specific type of DVD player thatthe user has, without requiring that the user is aware of any of thespecific details (such as the specific numerical model number that theypurchased). Instead, all the user needs to know is that they have a DVDplayer from a particular manufacturer (as opposed to the specific typeof DVD player).

FIG. 4 presents embodiments of a flow diagram illustrating method 400for displaying a user interface, which may be performed by a portableelectronic device, such as portable electronic device 110 (FIG. 1).During operation, the portable electronic device receives, via aninterface circuit in the portable electronic device, device-stateinformation (operation 410) from an A/V hub specifying a current stateof an entertainment device. Then, the portable electronic devicegenerates a user interface (operation 414) that includes one or morevirtual command icons associated with the current state and one or morerelated states of the entertainment device that are related to thecurrent state in a state diagram by corresponding operations thattransition the entertainment device from the current state to the one ormore related states. Next, the portable electronic device displays theuser interface on a touch-sensitive display (operation 416).

In some embodiments, the portable electronic device optionally receives,via the interface circuit, classification information for theentertainment device (which specifies a type of the entertainment deviceand/or a manufacturer of the entertainment device), and/or contextinformation (operation 412) that specifies a context of A/V contentdisplayed on the entertainment device. In these embodiments, theportable electronic device generates the user interface (operation 414)based on: the type of the entertainment device, the manufacturer of theentertainment device, and/or the context.

Moreover, the portable electronic device may optionally receive, via theinterface circuit, user-interface activity information (operation 418)specifying activation of a virtual command icon in the one or morevirtual command icons, where the activation of the virtual command iconspecifies a transition of the entertainment device from the currentstate to a new current state in the state diagram. For example, theactivation of the virtual command icon may involve a user of theportable electronic device contacting the touch-sensitive display withina strike area of the virtual command icon and then releasing thecontact. In response to receiving the user-interface activityinformation, the portable electronic device may optionally: modify theuser interface (operation 420) to change the one or more virtual commandicons based on the new current state; and display the modified userinterface (operation 420).

Embodiments of the feedback technique are further illustrated in FIG. 5,which presents a drawing illustrating communication between portableelectronic device 110 and A/V hub 112. In particular, processor 516 inA/V hub 112 may determine device-state information 518, such as acurrent state of entertainment device 512. For example, A/V hub 112 maydetermine device-state information 518 based on packets or framescommunicated 514 with entertainment device 512 via interface circuit510. Then, processor 516 may instruct interface circuit 510 towirelessly communicate device-state information 518 to portableelectronic device 110. In addition, processor 516 may instruct interfacecircuit 510 to wirelessly communicate additional information to portableelectronic device 110, such as: a type of entertainment device 512, amanufacturer of entertainment device 512, and/or context informationspecifying a context of A/V content displayed on entertainment device512.

After interface circuit 520 in portable electronic device 110 receivesdevice-state information 518, the type, the manufacturer, and/or thecontext information, processor 522 may generate user interface 524 (orinstructions for objects or graphical information in user interface 524)based on: a state diagram of entertainment device 512 (which may bestored in memory in portable electronic device 110), device-stateinformation 518, the type, the manufacturer, and/or the contextinformation. For example, processor 522 may generate instructions forthe objects or graphical information that includes or specifies userinterface 524, including display-specific information, such as: alocation where the graphical information is to be displayed ontouch-sensitive display 124 (TSD 124) having a particular type, displaysize, and/or an aspect ratio or geometry, e.g., an aspect ratio of 5:4or 4:3 with a display diagonal of at least 3.5 or 5 in. (These valuesare for purposes of illustration only, and a wide variety of displaysizes, aspect ratios and types may be used in touch-sensitive display124.) Furthermore, generating user interface 524 may involve calculatinga two or three-dimensional model and/or rendering operations, such as:two or three-dimensional projection, ray tracing, shading, coloring,texturing, illumination effects, texture mapping, and/or anti-aliasing.In the case of a three-dimensional touch-sensitive display 124, therendering operations may include calculating one or more images thatinclude or represent: image parallax, motion parallax (based on motionof the user relative to touch-sensitive display 124) and/or prehension(which may allow the user to perceive three-dimensional tactile orhaptic interaction with objects).

Then, processor 522 may display user interface 524 on touch-sensitivedisplay 124 via touch-screen I/O controller 526.

Subsequently, while a user is using portable electronic device 110,touch-screen I/O controller 526 may provide user-interface activityinformation 530 to processor 522 based on user interaction 528 withtouch-sensitive display 124, such as: the user making or breakingcontact with a surface of touch-sensitive display 124, moving a touchcontact point on the surface, etc. Then, processor 522 may instructinterface circuit 520 to communicate user-interface activity information530 to A/V hub 112.

After interface circuit 510 in A/V hub 112 receives user-interfaceactivity information 530, interface circuit 510 may communicateuser-interface activity information 530 to entertainment device 512.Then, A/V hub 112 may determine device-state information 534 based onpackets or frames communicated 532 with entertainment device 512 viainterface circuit 510. Next, processor 516 may instruct interfacecircuit 510 to wirelessly communicate device-state information 534 toportable electronic device 110.

Furthermore, after interface circuit 520 in portable electronic device110 receives device-state information 534, processor 522 may generateuser interface 536 (or instructions for objects or graphical informationin user interface 536) based on: the state diagram of entertainmentdevice 512, device-state information 536, the type, the manufacturer,and/or the context information. Next, processor 522 may display userinterface 536 on touch-sensitive display 124 via touch-screen I/Ocontroller 526. In this way, portable electronic device 110 and A/V hub112 may dynamically adapt the user interface as the current state ofentertainment device 512 changes.

Alternatively or additionally, as noted previously, user-interfaceinformation that specifies the user interface may be generated by theA/V hub. FIG. 6 presents embodiments of a flow diagram illustratingmethod 600 for providing a user interface, which may be performed by anA/V hub, such as A/V hub 112 (FIG. 1). During operation, the A/V hubdetermines, via a processor in the A/V hub, device-state information(operation 610), wherein the device-state information specifies acurrent state of an entertainment device. Then, the A/V hub generates,based on the determined device-state information, user-interfaceinformation that specifies a user interface (operation 612) thatincludes one or more virtual command icons associated with the currentstate and one or more related states of the entertainment device thatare related to the current state in a state diagram by correspondingoperations that transition the entertainment device from the currentstate to the one or more related states. Next, the A/V hub provides theuser-interface information (operation 614) to a portable electronicdevice for display on a touch-sensitive display in the portableelectronic device.

Note that the A/V hub may optionally generate the user-interfaceinformation based on: a type of the entertainment device, a manufacturerof the entertainment device, and/or context information that specifies acontext of A/V content displayed on the entertainment device. Thus, inthese embodiments, the one or more virtual command icons and/or theirlocations in the user interface may be based on: the type of theentertainment device, the manufacturer of the entertainment device,and/or the context.

Moreover, the A/V hub may optionally receive, via the interface circuit,user-interface activity information (operation 616) specifyingactivation of a virtual command icon in the one or more virtual commandicons, where the activation of the virtual command icon specifies atransition of the entertainment device from the current state to a newcurrent state in the state diagram. For example, the activation of thevirtual command icon may involve a user of the portable electronicdevice contacting the touch-sensitive display within a strike area ofthe virtual command icon and then releasing the contact. In response toreceiving the user-interface activity information, the A/V hub mayoptionally: modify the user-interface information (operation 618) tochange the one or more virtual command icons based on the new currentstate; and provide, via the interface circuit, the modifieduser-interface information (operation 620) for display on thetouch-sensitive display in the portable electronic device.

Embodiments of the feedback technique are further illustrated in FIG. 7,which presents a drawing illustrating communication between portableelectronic device 110 and A/V hub 112. In particular, processor 516 inA/V hub 112 may determine device-state information 518, such as acurrent state of entertainment device 512. For example, A/V hub 112 maydetermine device-state information 518 based on packets or framescommunicated 514 with entertainment device 512 via interface circuit510.

Then, processor 516 may generate user-interface information 710 thatspecifies user interface 524 (or instructions for objects or graphicalinformation in user interface 524) based on: a state diagram ofentertainment device 512, device-state information 518, a type ofentertainment device 512, a manufacturer of entertainment device 512,and/or context information. For example, processor 516 may generateinstructions for the objects or graphical information that includes orspecifies user interface 524, including display-specific information,such as: a location where the graphical information is to be displayedon touch-sensitive display 124 (TSD 124) having a particular type,display size, and/or an aspect ratio or geometry, e.g., an aspect ratioof 5:4 or 4:3, with a display diagonal of at least 3.5 or 5 in. (Thesevalues are for purposes of illustration only, and a wide variety ofdisplay sizes, aspect ratios and types may be used in touch-sensitivedisplay 124.) Furthermore, generating user-interface information 710 mayinvolve calculating a two or three-dimensional model and/or renderingoperations, such as: two or three-dimensional projection, ray tracing,shading, coloring, texturing, illumination effects, texture mapping,and/or anti-aliasing. In the case of a three-dimensional touch-sensitivedisplay 124, the rendering operations may include calculating one ormore images that include or represent: image parallax, motion parallax(based on motion of the user relative to touch-sensitive display 124)and/or prehension (which may allow the user to perceivethree-dimensional tactile or haptic interaction with objects).

Then, processor 516 may provide user-interface information 710 toportable electronic device 110. After user interface 520 receivesuser-interface information 710, processor 522 may display user interface524 on touch-sensitive display 124 via touch-screen I/O controller 526.

Subsequently, while a user is using portable electronic device 110,touch-screen I/O controller 526 may provide user-interface activityinformation 530 to processor 522 based on user interaction 528 withtouch-sensitive display 124, such as: the user making or breakingcontact with a surface of touch-sensitive display 124, moving a touchcontact point on the surface, etc. Then, processor 522 may instructinterface circuit 520 to communicate user-interface activity information530 to A/V hub 112.

After interface circuit 510 in A/V hub 112 receives user-interfaceactivity information 530, interface circuit 510 may communicateuser-interface activity information 530 to entertainment device 512.Then, A/V hub 112 may determine device-state information 534 based onpackets or frames communicated 532 with entertainment device 512 viainterface circuit 510. Next, processor 516 may generate user-interfaceinformation 712 that specifies user interface 536 (or instructions forobjects or graphical information in user interface 536) based on: thestate diagram of entertainment device 512, device-state information 534,the type, the manufacturer, and/or the context information.

Furthermore, processor 516 may instruct interface circuit 510 towirelessly communicate user-interface information 712 to portableelectronic device 110. After interface circuit 520 in portableelectronic device 110 receives user-interface information 712, processor522 may display user interface 536 on touch-sensitive display 124 viatouch-screen I/O controller 526. In this way, portable electronic device110 and A/V hub 112 may dynamically adapt the user interface as thecurrent state of entertainment device 512 changes.

FIG. 8 presents a flow diagram illustrating a method 800 forcommunicating a change in a state of an entertainment device, which maybe performed by a portable electronic device, such as portableelectronic device 110 (FIG. 1). During operation, the portableelectronic device (such as a cellular telephone or a remote control)receives user-interface activity information (operation 810) specifyingactivation of a virtual command icon in a user interface displayed on atouch-sensitive display in the portable electronic device, where theactivation of the virtual command icon specifies the change in the stateof an entertainment device (such as an A/V display device, e.g., atelevision). Then, the portable electronic device provides, via aninterface circuit in the portable electronic device, the user-interfaceactivity information (operation 812) to the A/V) hub. Moreover, theportable electronic device receives, via the interface circuit,device-state information (operation 814) from the A/V hub indicatingwhether the change in the state of the entertainment device hasoccurred. Next, when the change in the state of the entertainment devicehas not occurred (operation 816), the portable electronic deviceperforms, until the change in the state of the entertainment device hasoccurred, operations of: providing, via the interface circuit, theuser-interface activity information (operation 818) to the A/V hub; andreceiving, via the interface circuit, additional device-stateinformation (operation 820) from the A/V hub indicating whether thechange in the state of the entertainment device has occurred.

Note that, when the change in the state of the entertainment device hasnot occurred (operation 816), the portable electronic device mayoptionally provide, until the change in the state of the entertainmentdevice has occurred, feedback (operation 822) that indicates that thechange in the state of the entertainment device is being implemented.For example, providing the feedback (operation 822) may involvedisplaying a visual indicator in the user interface on thetouch-sensitive display that indicates that the change in the state ofthe entertainment device is being implemented. Alternatively oradditionally, providing the feedback (operation 822) may involveproviding, to the A/V hub, instructions for the visual indicator thatindicates that the change in the state of the entertainment device isbeing implemented, where the visual indicator is to be displayed on theentertainment device.

Furthermore, when the portable electronic device optionally receives theuser-interface activity information again (operation 824), such as whenthe user activates the virtual command icon in the user interface again,the portable electronic device may optionally provide sensory feedback(operation 826) that indicates that the change in the state of theentertainment device is being implemented. For example, the portableelectronic device may include a vibration actuator coupled to thecontrol circuit, and providing the sensory feedback may involveactivating the vibration actuator. However, in other embodiments anothertype of sensory feedback (such as a flashing light and/or sound) isprovided.

Alternatively or additionally to method 800, in some embodimentsoperations in the feedback technique are performed, at least in part, byA/V hub 112 (FIG. 1). This is shown in FIG. 9, which presents a flowdiagram illustrating a method 900 for communicating a change in a stateof an entertainment device, which may be performed by an A/V hub, suchas A/V hub 112 (FIG. 1). During operation, the A/V hub receives, via aninterface circuit, the user-interface activity information (operation910) from the portable electronic device specifying the activation ofthe virtual command icon in the user interface displayed on the portableelectronic device, where the activation of the virtual command iconspecifies the change in the state of an entertainment device. Then, theA/V hub provides the user-interface activity information (operation 912)to the entertainment device. Moreover, the A/V hub determines thedevice-state information (operation 914) about the entertainment deviceindicating whether the change in the state of the entertainment devicehas occurred. Next, when the change in the state of the entertainmentdevice has not occurred (operation 916), the A/V hub performs, until thechange in the state of the entertainment device has occurred, operationsof: providing the user-interface activity information (operation 918) tothe entertainment device; and determining additional device-stateinformation (operation 920) about the entertainment device indicatingwhether the change in the state of the entertainment device hasoccurred.

Note that, when the change in the state of the entertainment device hasnot occurred (operation 916), the A/V hub may optionally provide, untilthe change in the state of the entertainment device has occurred,feedback (operation 922) that indicates that the change in the state ofthe entertainment device is being implemented. For example, providingthe feedback (operation 922) may involve providing, via an interfacecircuit in the A/V hub, a visual indicator to the portable electronicdevice for display in the user interface to indicate that the change inthe state of the entertainment device is being implemented.Alternatively or additionally, providing the feedback (operation 922)may involve providing, to the entertainment device, the visual indicatorfor display on the entertainment device, where the visual indicatorindicates that the change in the state of the entertainment device isbeing implemented.

Furthermore, when the A/V hub optionally receives the user-interfaceactivity information again (operation 924) from the portable electronicdevice (such as when the user activates the virtual command icon in theuser interface again), the A/V hub may provide, via the interfacecircuit, a sensory-feedback instruction (operation 926) to the portableelectronic device that, when performed by the portable electronicdevice, indicates that the change in the state of the entertainmentdevice is being implemented. For example, the sensory-feedbackinstruction may instruct the portable electronic device to activate thevibration actuator. However, in other embodiments another type ofsensory feedback (such as a flashing light or sound) is provided.

FIG. 10 presents a drawing illustrating communication among theelectronic devices in FIG. 1, which presents a drawing illustratingcommunication between portable electronic device 110, A/V hub 112 andentertainment device 910. In particular, while a user is using portableelectronic device 110, touch-screen I/O controller 526 may provideuser-interface activity information 1012 to processor 522 based on userinteraction 1010 with touch-sensitive display (TSD 124), such as theuser making or breaking contact with a surface of touch-sensitivedisplay 124 (i.e., specifying activation of the virtual command icon inthe user interface that specifies the change in the state of theentertainment device). Then, processor 520 may instruct interfacecircuit 520 to communicate user-interface activity information 1012 toA/V hub 112.

After interface circuit 510 in A/V hub 112 receives user-interfaceactivity information 1012, interface circuit 510 may provideuser-interface activity information 1012 to entertainment device 512(such as A/V display device 114 or one of consumer-electronic devices116 in FIG. 1). Then, processor 516 may determine device-stateinformation 1016 indicating whether the change in the state ofentertainment device 512 has occurred based on subsequent communication1014 between A/V hub 112 and entertainment device 512.

In some embodiments, processor 516 optionally instructs interfacecircuit 510 to provide device-state information 1016 to portableelectronic device 110. After interface circuit 520 optionally receivesdevice-state information 1016, if the change in the state ofentertainment device 512 has not occurred, processor 1022 may optionallyperform, until the change in the state of entertainment device 512 hasoccurred, operations of: providing, via interface circuit 520,user-interface activity information 1012 to A/V hub 112; and receiving,via interface circuit 520, additional device-state information 1020 fromA/V hub 112 indicating whether the change in the state of theentertainment device has occurred. Similarly, when the change in thestate of entertainment device 512 has not occurred, processor 516 mayoptionally perform, until the change in the state of entertainmentdevice 512 has occurred, operations of: providing, via interface circuit510, user-interface activity information 1012 to entertainment device512; and determining additional device-state information 1020 aboutentertainment device 512 indicating whether the change in the state ofentertainment device 512 has occurred (based on communication 1018between A/V hub 112 and entertainment device 512). Note that A/V hub 112may only provide the user-interface activity information differentially,so that entertainment device 512 may only receive a change to theuser-interface activity information once.

However, because the communication between portable electronic device110 and A/V hub 112 is bidirectional, the situational awareness aboutthe state of entertainment device 512 can be used to keep the userinformed about the status of the user's instruction to change the stateof entertainment device 512 and to prevent the user from becomingfrustrated. In particular, portable electronic device 110 and/or A/V hub112 may provide feedback to the user about the status of the change inthe state of entertainment device 512.

For example, when the change in the state of entertainment device 512has not occurred, processor 522 may optionally provide, until the changein the state of entertainment device 512 has occurred, feedback thatindicates that the change in the state of entertainment device 512 isbeing implemented. In particular, processor 522 may optionally generatevisual indicator 1022 (such as instructions for graphical information oran object that indicates that the change in the state of entertainmentdevice 512 is being implemented), which is provided to touch-screen I/Ocontroller 526 and then displayed on TSD 124. Alternatively oradditionally, processor 522 may optionally instruct interface circuit520 to provide visual indicator 1022 to A/V hub 112. This visualindicator may be received by interface circuit 510 and forwarded toentertainment device 510 (or an A/V display device) for display to theuser. Note that if touch-screen I/O controller 526 subsequently providesuser-interface activity information 1026 to processor 522 based on userinteraction 1024 with TSD 124 (such as when the user activates thevirtual command icon in the user interface again), processor 522 mayoptionally provide an instruction for sensory feedback 1028 to asensory-feedback mechanism (not shown) (such as a light, a speakerand/or an eccentric-rotating-mass actuator or a linear-resonantactuator) that indicates the change in the state of entertainment device512 is being implemented (so that the user does not continue to activatethe virtual command icon in frustration). Alternatively or additionally,processor 522 may optionally instruct interface circuit 520 to provideuser-interface activity information 1026 to A/V hub 112.

Similarly, when the change in the state of entertainment device 512 hasnot occurred, processor 516 may optionally provide, until the change inthe state of entertainment device 512 has occurred, feedback thatindicates that the change in the state of entertainment device 512 isbeing implemented. For example, processor 512 may optionally generatevisual indicator 1022 (such as the instructions for the graphicalinformation or the object that indicates that the change in the state ofentertainment device 512 is being implemented), which is optionallyprovided, via interface circuit 510, to portable electronic device 110for display on TSD 124 and/or to entertainment device 512 for display tothe user. Note that if interface circuit 510 subsequently receivesuser-interface activity information 1026, processor 516 may optionallythe instruction for sensory feedback 1028 to portable electronic device110. This instruction for sensory feedback 1028 may optionally beperformed by the sensory-feedback mechanism (not shown).

Note that generating visual indicator 1022 (or instructions for visualindicator 1022) based on user-interface activity information 1012 or1026 and/or device-state information 1016 or 1020 may involve generatinginstructions for an object or graphical information that includes orspecifies visual feedback, such as a partially transparent graphicaloverlay that can be displayed on portable electronic device 110 and/orA/V display device 114. The graphical information may includedisplay-specific information, such as: a location where the graphicalinformation is to be displayed on a display having a particular type,display size, and/or an aspect ratio or geometry, e.g., an aspect ratioof 5:4 or 4:3 with a display diagonal of at least 3.5 or 5 in., or anaspect ratio of 16:9 with a display diagonal of at least 50 in. (Thesevalues are for purposes of illustration only, and a wide variety ofdisplay sizes, aspect ratios and types may be used in portableelectronic device 110 and/or A/V display device 114.) Furthermore,generating visual indicator 1022 may involve calculating a two orthree-dimensional model (such as a model of the virtual icons in theuser interface, which may be based on the state diagram) and/orrendering operations, such as: two or three-dimensional projection, raytracing, shading, coloring, texturing, illumination effects, texturemapping, and/or anti-aliasing. In the case of a three-dimensionaldisplay in A/V display device 114, the rendering operations may includecalculating one or more images that include or represent: imageparallax, motion parallax (based on motion of the user relative to A/Vdisplay device 114) and/or prehension (which may allow the user toperceive three-dimensional tactile or haptic interaction with objects).For example, visual indicator 1022 may include graphical information,such as: flashing a representation of the virtual command icon, changinga line thickness in the virtual command icon and/or adding a graphicalsymbol (such as an hour glass or a watch face). When displayed on A/Vdisplay device 114, visual indicator 1022 may include a graphicalrepresentation of the user interface, including locations or a layout ofone or more virtual command icons.

In this way, the feedback technique may leverage knowledge of thedevice-state information to keep the user information about the statusof their instruction to change the state of the entertainment device. Inthe process, the feedback technique may reduce user frustration, andthus may improve user satisfaction when using the portable electronicdevice, the A/V hub and/or the entertainment device.

Consequently, methods 200 (FIG. 2), 400 (FIG. 4), 600 (FIG. 6) 800 (FIG.8) and/or 900 (FIG. 9) may reduce user errors or mistakes when using theuser interface, which may improve the user experience when using theportable electronic device and/or the A/V hub.

In some embodiments of methods 200 (FIG. 2), 400 (FIG. 4), 600 (FIG. 6)800 (FIG. 8) and/or 900 (FIG. 9), there may be additional or feweroperations. For example, in a more general version of method 200 (FIG.2), when the entertainment device is electrically coupled to the inputconnector, the state-detection circuit may determine a state of theentertainment device and/or an identify the entertainment device.Moreover, the order of the operations may be changed, and/or two or moreoperations may be combined into a single operation.

As noted previously, the device-state information (such as whether anentertainment device is: electrically coupled to A/V hub 112 in FIG. 1,in a power-on state, in a power-off state, and/or another state, e.g., aplayback state, a pause state, a stop state, etc.) may be determinedusing hardware (such as a state-detection circuit) and/or software(which may be executed by a processor and, more generally, a controlcircuit). FIG. 11 presents a block diagram illustrating astate-detection circuit 1110 in A/V hub 112 (FIG. 1). In A/V hub 112(FIG. 1), input connector 1112 (which may be compatible with an HDMIstandard) may be electrically coupled to an entertainment device.State-detection circuit 1110 may be coupled to at least pin 1114 ininput connector 1112, so that, when the entertainment device iselectrically coupled to input connector 1112, state-detection circuit1110 establishes a ground loop between A/V hub 112 (FIG. 1) and theentertainment device. For example, pin 1114 may include a transitionminimized differential signaling (TMDS) data1 shield. (Alternatively,pin 1114 may include a TMDS data1 shield.) Moreover, state-detectioncircuit 1110 may include: an energy-dissipation component (such asresistor 1116) electrically coupled to a power-supply voltage and pin1114 (which may provide electrostatic-discharge protection); anenergy-storage component (such as capacitor 1118) electrically coupledto pin 1114 and ground; and a bi-directional voltage clamp (such asvaristor 1120 or a Verner diode), in parallel with capacitor 1118,electrically coupled to pin 1114 and ground. For example, resistor 1116may be 150 kΩ and capacitor 1118 may be 0.047 g. In some embodiments,state-detection circuit 1110 includes a general-purpose input/output(GPIO) device 1126 coupled to pin 1114. The behavior (such as an inputpin, an output pin, enabled or disabled) of GPIO device 1126 may becontrolled using control signals or instructions from control logic1124.

FIG. 12 presents a flow diagram illustrating a method 1200 for detectingan entertainment device, which may be performed by an A/V hub (such asA/V hub 112 in FIG. 1) using state-detection circuit 1110 in FIG. 11.During operation, a control circuit (such as a processor and/or controllogic 1124, which may be included in or external to state-detectioncircuit 1110) in A/V hub 112 (FIG. 1), which is electrically coupled toinput connector 1112, detects whether there is electrical couplingbetween the entertainment device and input connector 1112 usingstate-detection circuit 1110 (FIG. 11). In particular, detecting whetherthere is electrical coupling between the entertainment device and inputconnector 1112 may involve: setting pin 1114 as an input (operation1210), where pin 1114 is then pulled to a power-supply voltage bycontrol logic 1124; measuring a voltage on pin 1114 (operation 1212)using control logic 1124; and detecting the electrical coupling betweenthe entertainment device and input connector 1112 when the voltage onpin 1114 is less than or equal to a predefined value (operation 1214),such as when the voltage is approximately ground, using control logic1124. Note that, when the electrical coupling between the entertainmentdevice and input connector 1112 is detected, control logic 1124 may: setpin 1114 as an output and electrically couple pin 1114 to ground(operation 1218), which may improve signal integrity; and measure asecond voltage (operation 1220) on hotplug-detect pin 1122 in inputconnector 1112. When the second voltage on hotplug-detect pin 1122 isless than or equal to the predefined value (operation 1222), controllogic 1124 may set pin 1114 as an input (operation 1224) and repeat themeasurement of the voltage on pin 1114 (operation 1212). Alternatively,when the voltage equals or exceeds a second predefined value (operation1226), such as when the voltage is approximately the power-supplyvoltage, control logic 1124 may repeat detecting whether there iselectrical coupling between the entertainment device and input connector1112. Furthermore, when the voltage is less than or equal to thepredefined value (operation 1214), control logic 1124 may identify acurrent state (operation 1216) of the entertainment device, such as: apower-off state, and a standby state. For example, control logic 1124may provide the set of first control commands, provide the set of secondcontrol commands, and/or may monitor (via one or more pins in inputconnector 1112) content activity, such as a data stream to and/or fromthe entertainment device. Thus, control logic 1124 may determine thatthe entertainment device is: in the power-off state when there is no adata stream; in the standby state when the data stream has a low datarate; and in the playback state when the data stream has a data rateassociated with A/V content and/or includes the A/V content. Note that,when the second voltage on hotplug-detect pin 1122 is less than or equalto the predefined value (operation 1222) and when the voltage is lessthan or equal to the predefined value (operation 1214), control logic1124 may repeat setting pin 1114 as the output and electrically couplingpin 1114 to ground (operation 1218).

When the electrical coupling between the entertainment device and inputconnector 1112 is detected, control logic 1124 may optionally attempt toidentify the entertainment device by providingconsumer-electronics-control commands (which may be compatible with anHDMI standard) to the entertainment device. Alternatively oradditionally (such as when the attempt is unsuccessful), control logic1124 may provide a set of first control commands associated withdifferent types of entertainment devices until, in response, contentactivity (such as packets or frames associated with a data stream ofcontent communicated to and/or from the entertainment device) isdetected by control logic 1124 via input connector 1112. For example,the set of first commands may include: a play command for the differenttypes of entertainment devices; and/or a trick-mode command (such asfast forward, reverse, fast reverse, or skip) for the different types ofentertainment devices. Moreover, when the content activity is detected,control logic 1124 may provide a set of second control commandsassociated with different providers of entertainment devices until achange in a state of the entertainment device is detected by controllogic 1124 via input connector 1112 and state-detection circuit 1110.The set of second control commands may include: power-on controlcommands for the different providers of entertainment devices; and/orpower-off control commands for the different providers of entertainmentdevices.

Alternatively or additionally, during operation control logic 1124 maydetect whether there is electrical coupling between the entertainmentdevice and input connector 1112 using state-detection circuit 1110 (FIG.11). When the electrical coupling between the entertainment device andinput connector 1112 is detected, control logic 1124 may: set pin 1114as an output and electrically couple pin 1114 to ground; and measure thesecond voltage on hotplug-detect pin 1122 in input connector 1112. Whenthe second voltage on hotplug-detect pin 1122 is less than or equal tothe predefined value, control logic 1124 may set pin 1114 as an inputand measure a voltage on pin 1114. Moreover, when the voltage equals orexceeds the second predefined value, control logic 1124 may repeatdetecting whether there is electrical coupling between the entertainmentdevice and input connector 1112. Furthermore, when the voltage is lessthan or equal to the predefined value, control logic 1124 may identifythe current state of the entertainment device. In some embodiments,control logic 1124: provides a control command to the entertainmentdevice; and identifies an additional state of the entertainment devicebased on content (such as A/V content) that is provided and/or receivedby the entertainment device in response to the control command. Forexample, the control command may include: a play command, and/or atrick-mode command (such as fast forward or fast skip, slow forward orslow skip, fast reverse, or slow reverse).

FIG. 13 presents a drawing illustrating a state diagram 1300 of a givenentertainment device in FIG. 1. This state diagram specifies theavailable commands in different menus for the given entertainment devicethat are relevant based on a current state of the given entertainmentdevice. (Examples of the sets of commands in menus for differententertainment devices are shown in Table 1.) Thus, based on thedevice-state information, particular subsets of the available commandsmay be included or displayed in a user interface, such as a subset ofthe available commands that are relevant next states that can beaccessed by the user (by activating a virtual command icon, e.g., bycontacting a surface of a touch-sensitive display within a strike areaassociated with the virtual command icon, and then breaking contact withthe surface of the touch-sensitive display) from the current state ofthe given entertainment device.

TABLE 1 Menu 1 Menu 2 Menu 3 Device Commands Commands Commands Cable/DVRVolume Record Live TV up/down Mute Numbers Info DVR — Input Channel —Menu up/down Guide — Zoom Play/Pause — Exit Fast — Help forward/rewindPrev/last — Clear Cable Volume — — up/down Mute — — Guide — — Channel —— up/down Record — — DVR Volume — — up/down Mute — — Play/Pause — — Fast— — forward/rewind Channel — — up/down Streaming Volume — — up/down Mute— — D-pad — — Home — — Play/Pause — — Fast — — forward/rewind DVD Volume— — up/down Mute — — Play/Pause — — Fast — — forward/rewind Chapter — —skip/prev Menu — — D-Pad — — Chapter select — — Receiver Volume — —up/down Mute — — Menu — — D-pad — — CD player/MP3 Volume — — player dockup/down Mute — — Source — — D-pad — — Mute — — Play — — Guide — — GamingVolume — — up/down Mute — — D-pad — — Select — — Back — — Transport — —TV Volume — — up/down Mute — — D-pad — — Channel — — up/down Apps — —User Online Volume — — Content up/down Mute — — D-pad — — Other Volume —— up/down Mute — — D-pad — — Channel — — up/down Apps — — ComputerVolume — — up/down Mute — — D-pad — — Mouse — —

For example, if the device-state information indicates that the deviceis in a power-off state, the user interface may initially only includeone virtual command icon, which, when activated by a user, may power onthe given entertainment device. This is shown in FIG. 14, which presentsa drawing illustrating a user interface 1400 on a touch-sensitivedisplay 124 in portable electronic device 110 in FIG. 1. In particular,user interface 1400 includes a virtual command icon 1410 correspondingto a power-on command. In this case, strike or contact area 1412associated with virtual command icon 1410 may be much larger (e.g., aslarge as user interface 1400).

After virtual command icon 1410 has been activated by the user, the userinterface may include one or more virtual command icons that areassociated with the current state and one or more related states of theentertainment device that are related to the current state in a statediagram by corresponding operations that transition the entertainmentdevice from the current state to the one or more related states.Furthermore, the user interface may exclude one or more additionalvirtual command icons associated with one or more unrelated states ofthe entertainment device that are unrelated to the current state bydirect transitions in the state diagram. Thus, as shown in FIG. 15,which presents a drawing illustrating a user interface 1500 on atouch-sensitive display 124 in portable electronic device 110 in FIG. 1,after the given entertainment device is powered on, user interface 1500may include a virtual command icon 1510 corresponding to a play commandand a virtual command icon 1512 that, when activated, powers off thegiven entertainment device. However, a virtual command iconcorresponding to a stop command may not be displayed because it is notrelevant to the current state of the entertainment device.

In some embodiments, a given operation directly transitions the givenentertainment device from the current state to one of the additionalstates without passing through an intermediate state in the statediagram. These operations, and their associated states, may be the onesthat are relevant at any given time for inclusion in the user interface.

While power-off and power-on were used as illustration of states of thegiven entertainment device, in other embodiments the current stateincludes one of a wide variety of states that are different than thepower-off state or the power-on state.

For example, as shown in FIG. 16, which presents a drawing illustratinga user interface 1600 on a touch-sensitive display 124 in portableelectronic device 110 in FIG. 1, after virtual command icon 1510 (FIG.15) has been activated by the user, user interface 1600 may include: avirtual command icon 1610 corresponding to a fast-forward command, avirtual command icon 1612 corresponding to a slow-forward command, avirtual command icon 1614 corresponding to a stop command, a virtualcommand icon 1616 corresponding to a slow-reverse command, a virtualcommand icon 1618 corresponding to a fast-reverse command, and a virtualcommand icon 1620 corresponding to a main menu.

FIG. 17-20 present drawings of user interfaces on a touch-sensitivedisplay 124 in portable electronic device 110 in FIG. 1. In particular,FIG. 17 presents a user interface 1700 that can be used to select aparticular entertainment device, such as a DVR. When the DVR isselected, the appropriate set of virtual command icons may be displayedbased on the current state of the DVR. This is shown in FIG. 18, whichpresents user interface 1800. Similarly, when user interface 1900 inFIG. 19 is used to select a Blu-ray player, user interface 2000 in FIG.20 with the appropriate set of virtual command icons may be displayed.

We now describe embodiments of an electronic device. FIG. 21 presents ablock diagram illustrating an electronic device 2100, such as portableelectronic device 110, A/V hub 112 or A/V display device 114 in FIG. 1.This electronic device includes processing subsystem 2110, memorysubsystem 2112, networking subsystem 2114 and optional feedbacksubsystem 2134. Processing subsystem 2110 includes one or more devicesconfigured to perform computational operations. For example, processingsubsystem 2110 can include one or more microprocessors,application-specific integrated circuits (ASICs), microcontrollers,programmable-logic devices, and/or one or more digital signal processors(DSPs). One or more of these components in processing subsystem aresometimes referred to as a ‘control circuit.’

Memory subsystem 2112 includes one or more devices for storing dataand/or instructions for processing subsystem 2110 and networkingsubsystem 2114. For example, memory subsystem 2112 can include dynamicrandom access memory (DRAM), static random access memory (SRAM), and/orother types of memory. In some embodiments, instructions for processingsubsystem 2110 in memory subsystem 2112 include: one or more programmodules or sets of instructions (such as program module 2122 oroperating system 2124), which may be executed by processing subsystem2110. Note that the one or more program modules may constitute acomputer-program mechanism, such as a computer program or software.Moreover, instructions in the various modules in memory subsystem 2112may be implemented in: a high-level procedural language, anobject-oriented programming language, and/or in an assembly or machinelanguage. Furthermore, the programming language may be compiled orinterpreted, e.g., configurable or configured (which may be usedinterchangeably in this discussion), to be executed by processingsubsystem 2110.

In addition, memory subsystem 2112 can include mechanisms forcontrolling access to the memory. In some embodiments, memory subsystem2112 includes a memory hierarchy that comprises one or more cachescoupled to a memory in electronic device 2100. In some of theseembodiments, one or more of the caches is located in processingsubsystem 2110.

In some embodiments, memory subsystem 2112 is coupled to one or morehigh-capacity mass-storage devices (not shown). For example, memorysubsystem 2112 can be coupled to a magnetic or optical drive, asolid-state drive, or another type of mass-storage device. In theseembodiments, memory subsystem 2112 can be used by electronic device 2100as fast-access storage for often-used data, while the mass-storagedevice is used to store less frequently used data.

Networking subsystem 2114 includes one or more devices configured tocouple to and communicate on a wired and/or wireless network (i.e., toperform network operations), including: control logic 2116, interfacecircuits 2118 and associated antennas 2120. (While FIG. 21 includesantennas 2120, in some embodiments electronic device 2100 includes oneor more nodes, such as nodes 2108, e.g., pads, which can be coupled toantennas 2120. Thus, electronic device 2100 may or may not includeantennas 2120.) For example, networking subsystem 2114 can include aBluetooth networking system, a cellular networking system (e.g., a 3G/4Gnetwork such as UMTS, LTE, etc.), a universal serial bus (USB)networking system, a networking system based on the standards describedin IEEE 802.11 (e.g., a Wi-Fi networking system), an Ethernet networkingsystem, and/or another networking system. Note that the combination of agiven one of interface circuits 2118 and at least one of antennas 2120may constitute a radio. In some embodiments, networking subsystem 2114includes a wired interface, such as HDMI interface 2130 (which mayinclude a state-detection circuit).

Networking subsystem 2114 includes processors, controllers,radios/antennas, sockets/plugs, and/or other devices used for couplingto, communicating on, and handling data and events for each supportednetworking system. Note that mechanisms used for coupling to,communicating on, and handling data and events on the network for eachnetwork system are sometimes collectively referred to as a ‘networkinterface’ for the network system. Moreover, in some embodiments a‘network’ between the electronic devices does not yet exist. Therefore,electronic device 2100 may use the mechanisms in networking subsystem2114 for performing simple wireless communication between the electronicdevices, e.g., transmitting advertising or beacon frames and/or scanningfor advertising frames transmitted by other electronic devices asdescribed previously.

Within electronic device 2100, processing subsystem 2110, memorysubsystem 2112, networking subsystem 2114 and optional feedbacksubsystem 2134 are coupled together using bus 2128. Bus 2128 may includean electrical, optical, and/or electro-optical connection that thesubsystems can use to communicate commands and data among one another.Although only one bus 2128 is shown for clarity, different embodimentscan include a different number or configuration of electrical, optical,and/or electro-optical connections among the subsystems.

In some embodiments, electronic device 2100 includes a display subsystem2126 for displaying information on a display (such as the communicationwarning message), which may include a display driver, an I/O controllerand the display. Note that a wide variety of display types may be usedin display subsystem 2126, including: a two-dimensional display, athree-dimensional display (such as a holographic display or a volumetricdisplay), a head-mounted display, a retinal-image projector, a heads-updisplay, a cathode ray tube, a liquid-crystal display, a projectiondisplay, an electroluminescent display, a display based on electronicpaper, a thin-film transistor display, a high-performance addressingdisplay, an organic light-emitting diode display, a surface-conductionelectronic-emitter display, a laser display, a carbon-nanotube display,a quantum-dot display, an interferometric modulator display, amulti-touch touchscreen (which is sometimes referred to as atouch-sensitive display), and/or a display based on another type ofdisplay technology or physical phenomenon.

Furthermore, optional feedback subsystem 2134 may include one or moresensor-feedback mechanisms or devices, such as: a vibration mechanism ora vibration actuator (e.g., an eccentric-rotating-mass actuator or alinear-resonant actuator), a light, one or more speakers, etc., whichcan be used to provide feedback to a user of electronic device 2100(such as sensory feedback about the status of a user instruction tochange the state of one of the components in system 100 in FIG. 1).

Electronic device 2100 can be (or can be included in) any electronicdevice with at least one network interface. For example, electronicdevice 2100 can be (or can be included in): a desktop computer, a laptopcomputer, a subnotebook/netbook, a server, a tablet computer, asmartphone, a cellular telephone, a consumer-electronic device (such asa television, a set-top box, audio equipment, video equipment, etc.), aremote control, a portable computing device, an access point, a router,a switch, communication equipment, test equipment, and/or anotherelectronic device.

Although specific components are used to describe electronic device2100, in alternative embodiments, different components and/or subsystemsmay be present in electronic device 2100. For example, electronic device2100 may include one or more additional processing subsystems, memorysubsystems, networking subsystems, and/or display subsystems. Moreover,while one of antennas 2120 is shown coupled to a given one of interfacecircuits 2118, there may be multiple antennas coupled to the given oneof interface circuits 2118. For example, an instance of a 3×3 radio mayinclude three antennas. Additionally, one or more of the subsystems maynot be present in electronic device 2100. Furthermore, in someembodiments, electronic device 2100 may include one or more additionalsubsystems that are not shown in FIG. 21. Also, although separatesubsystems are shown in FIG. 21, in some embodiments, some or all of agiven subsystem or component can be integrated into one or more of theother subsystems or component(s) in electronic device 2100. For example,in some embodiments program module 2122 is included in operating system2124.

Moreover, the circuits and components in electronic device 2100 may beimplemented using any combination of analog and/or digital circuitry,including: bipolar, PMOS and/or NMOS gates or transistors. Furthermore,signals in these embodiments may include digital signals that haveapproximately discrete values and/or analog signals that have continuousvalues. Additionally, components and circuits may be single-ended ordifferential, and power supplies may be unipolar or bipolar.

An integrated circuit may implement some or all of the functionality ofnetworking subsystem 2114, such as one or more radios. Moreover, theintegrated circuit may include hardware and/or software mechanisms thatare used for transmitting wireless signals from electronic device 2100and receiving signals at electronic device 2100 from other electronicdevices. Aside from the mechanisms herein described, radios aregenerally known in the art and hence are not described in detail. Ingeneral, networking subsystem 2114 and/or the integrated circuit caninclude any number of radios.

In some embodiments, networking subsystem 2114 and/or the integratedcircuit include a configuration mechanism (such as one or more hardwareand/or software mechanisms) that configures the radios to transmitand/or receive on a given channel (e.g., a given carrier frequency). Forexample, in some embodiments, the configuration mechanism can be used toswitch the radio from monitoring and/or transmitting on a given channelto monitoring and/or transmitting on a different channel. (Note that‘monitoring’ as used herein comprises receiving signals from otherelectronic devices and possibly performing one or more processingoperations on the received signals, e.g., determining if the receivedsignal comprises an advertising frame, calculating a performance metric,performing spectral analysis, etc.) Furthermore, networking subsystem2114 may include at least one port (such as an HDMI port 2132) toreceive and/or provide the information in the data stream to A/V displaydevice 114 (FIG. 1) and/or one of the one or more consumer-electronicdevices 116 (FIG. 1).

While a communication protocol compatible with Wi-Fi was used as anillustrative example, the described embodiments may be used in a varietyof network interfaces. Furthermore, while some of the operations in thepreceding embodiments were implemented in hardware or software, ingeneral the operations in the preceding embodiments can be implementedin a wide variety of configurations and architectures. Therefore, someor all of the operations in the preceding embodiments may be performedin hardware, in software or both. For example, at least some of theoperations in the feedback technique may be implemented using programmodule 2122, operating system 2124 (such as drivers for interfacecircuits 2118) and/or in firmware in interface circuits 2118.Alternatively or additionally, at least some of the operations in thefeedback technique may be implemented in a physical layer, such ashardware in interface circuits 2118.

Moreover, while the preceding embodiments included a touch-sensitivedisplay in the portable electronic device that the user touches (e.g.,with a finger or digit, or a stylus), in other embodiments the userinterface is display on a display in the portable electronic device andthe user interacts with the user interface without making contact ortouching the surface of the display. For example, the user's interact(s)with the user interface may be determined using time-of-flightmeasurements, motion sensing (such as a Doppler measurement) or anothernon-contact measurement that allows the position, direction of motionand/or speed of the user's finger or digit (or a stylus) relative toposition(s) of one or more virtual command icons to be determined. Inthese embodiments, note that the user may activate a given virtualcommand icon by performing a gesture (such as ‘tapping’ their finger inthe air without making contact with the surface of the display). In someembodiments, the user navigates through the user interface and/oractivates/deactivates functions of one of the components in system 100(FIG. 1) using spoken commands or instructions (i.e., via voicerecognition) and/or based on where they are looking in the visualfeedback displayed on A/V display device 114 in FIG. 1 (e.g., bytracking the user's gaze or where the user is looking).

Furthermore, while A/V hub 112 (FIG. 1) was illustrated as a separatecomponent from A/V display device 114 (FIG. 1), in some embodiments thecomponents are combined into a single component or a single electronicdevice.

While the preceding embodiments illustrated the feedback technique withaudio and video content, in other embodiments the feedback technique isused in the context of an arbitrary type of data or information. Forexample, the feedback technique may be used with home-automation data.In these embodiments, A/V hub 112 (FIG. 1) may facilitate communicationamong and control of a wide variety of electronic devices, includingelectronic devices in addition to or other than entertainment devices.Thus, A/V hub 112 (FIG. 1) and the feedback technique may be used tofacilitate or implement the so-called Internet of things.

Moreover, while HDMI is used as an illustrative example of a contentformat in the preceding discussion, in other embodiments content that iscompatible with another format or standard is used in the embodiments ofthe feedback technique.

In the preceding description, we refer to ‘some embodiments.’ Note that‘some embodiments’ describes a subset of all of the possibleembodiments, but does not always specify the same subset of embodiments.

The foregoing description is intended to enable any person skilled inthe art to make and use the disclosure, and is provided in the contextof a particular application and its requirements. Moreover, theforegoing descriptions of embodiments of the present disclosure havebeen presented for purposes of illustration and description only. Theyare not intended to be exhaustive or to limit the present disclosure tothe forms disclosed. Accordingly, many modifications and variations willbe apparent to practitioners skilled in the art, and the generalprinciples defined herein may be applied to other embodiments andapplications without departing from the spirit and scope of the presentdisclosure. Additionally, the discussion of the preceding embodiments isnot intended to limit the present disclosure. Thus, the presentdisclosure is not intended to be limited to the embodiments shown, butis to be accorded the widest scope consistent with the principles andfeatures disclosed herein.

What is claimed is:
 1. An audio/video (A/V) hub, comprising: an antenna;an interface circuit, coupled to the antenna, configured to communicatewith a portable electronic device; and a control circuit, coupled to theinterface circuit, configured to: receive, via the interface circuit,user-interface activity information from the portable electronic devicespecifying activation of a virtual command icon in a user interfacedisplayed on the portable electronic device, wherein the activation ofthe virtual command icon specifies a change in a state of anentertainment device; provide the user-interface activity information tothe entertainment device; determine device-state information about theentertainment device indicating whether the change in the state of theentertainment device has occurred; and when the change in the state ofthe entertainment device has not occurred, perform, until the change inthe state of the entertainment device has occurred, operations of:providing the user-interface activity information to the entertainmentdevice; and determining additional device-state information about theentertainment device indicating whether the change in the state of theentertainment device has occurred.
 2. The A/V hub of claim 1, wherein,when the change in the state of the entertainment device has notoccurred, the control circuit is configured to provide, until the changein the state of the entertainment device has occurred, feedback thatindicates that the change in the state of the entertainment device isbeing implemented.
 3. The A/V hub of claim 2, wherein providing thefeedback involves providing, via the interface circuit, a visualindicator to the portable electronic device for display in the userinterface to indicate that the change in the state of the entertainmentdevice is being implemented.
 4. The A/V hub of claim 2, whereinproviding the feedback involves providing, to the entertainment device,a visual indicator for display on the entertainment device; and whereinthe visual indicator indicates that the change in the state of theentertainment device is being implemented.
 5. The A/V hub of claim 1,wherein, when the control circuit receives the user-interface activityinformation again from the portable electronic device, the controlcircuit is configured to provide, via the interface circuit, asensory-feedback instruction to the portable electronic device that,when performed by the portable electronic device, indicates that thechange in the state of the entertainment device is being implemented. 6.The A/V hub of claim 5, wherein the sensory-feedback instructioninstructs the portable electronic device to activate a vibrationactuator.
 7. The A/V hub of claim 1, wherein the control circuitcomprises: memory configured to store a program module; and a processor,coupled to the interface circuit and the memory, configured to executethe program module, wherein the program module includes instructions foroperations performed by the control circuit.
 8. The A/V hub of claim 1,wherein entertainment device includes a television.
 9. A non-transitorycomputer-program product for use in conjunction with an audio/video(A/V) hub, the computer-program product comprising a computer-readablestorage medium that stores a computer-program mechanism that, whenexecuted by the A/V hub, causes the A/V hub to communicate a change in astate of an entertainment device, the computer-program mechanismincluding: instructions for receiving, via an interface circuit in theA/V hub, user-interface activity information from a portable electronicdevice specifying activation of a virtual command icon in a userinterface displayed on the portable electronic device, wherein theactivation of the virtual command icon specifies a change in a state ofan entertainment device; instructions for providing the user-interfaceactivity information to the entertainment device; instructions fordetermining device-state information about the entertainment deviceindicating whether the change in the state of the entertainment devicehas occurred; and instructions for performing, when the change in thestate of the entertainment device has not occurred and until the changein the state of the entertainment device has occurred, operations of:providing the user-interface activity information to the entertainmentdevice; and determining additional device-state information about theentertainment device indicating whether the change in the state of theentertainment device has occurred.
 10. The computer-program product ofclaim 9, wherein the computer-program mechanism further comprisesinstructions for providing, when the change in the state of theentertainment device has not occurred and until the change in the stateof the entertainment device has occurred, feedback that indicates thatthe change in the state of the entertainment device is beingimplemented.
 11. The computer-program product of claim 10, whereinproviding the feedback involves providing, via the interface circuit, avisual indicator to the portable electronic device for display in theuser interface to indicate that the change in the state of theentertainment device is being implemented.
 12. The computer-programproduct of claim 10, wherein providing the feedback involves providing,to the entertainment device, a visual indicator for display on theentertainment device; and wherein the visual indicator indicates thatthe change in the state of the entertainment device is beingimplemented.
 13. The computer-program product of claim 9, wherein thecomputer-program mechanism further comprises instructions for providing,when the user-interface activity information is received again from theportable electronic device, a sensory-feedback instruction to theportable electronic device that, when performed by the portableelectronic device, indicates that the change in the state of theentertainment device is being implemented.
 14. The computer-programproduct of claim 13, wherein the sensory-feedback instruction instructsthe portable electronic device to activate a vibration actuator.
 15. Thecomputer-program product of claim 9, wherein entertainment deviceincludes a television.
 16. An audio/video (A/V) hub-implemented methodfor communicating a change in a state of an entertainment device,wherein the method comprises: receiving, via an interface circuit in theA/V hub, user-interface activity information from a portable electronicdevice specifying activation of a virtual command icon in a userinterface displayed on the portable electronic device, wherein theactivation of the virtual command icon specifies a change in a state ofan entertainment device; providing the user-interface activityinformation to the entertainment device; determining device-stateinformation about the entertainment device indicating whether the changein the state of the entertainment device has occurred; and using the A/Vhub, performing, when the change in the state of the entertainmentdevice has not occurred and until the change in the state of theentertainment device has occurred, operations of: providing theuser-interface activity information to the entertainment device; anddetermining additional device-state information about the entertainmentdevice indicating whether the change in the state of the entertainmentdevice has occurred.
 17. The method of claim 16, wherein the methodfurther comprises, when the change in the state of the entertainmentdevice has not occurred and until the change in the state of theentertainment device has occurred, providing feedback that indicatesthat the change in the state of the entertainment device is beingimplemented.
 18. The method of claim 17, wherein providing the feedbackinvolves providing, via the interface circuit, a visual indicator to theportable electronic device for display in the user interface to indicatethat the change in the state of the entertainment device is beingimplemented.
 19. The method of claim 17, wherein providing the feedbackinvolves providing, to the entertainment device, a visual indicator fordisplay on the entertainment device; and wherein the visual indicatorindicates that the change in the state of the entertainment device isbeing implemented.
 20. The method of claim 16, wherein the methodfurther comprises providing, when the user-interface activityinformation is received again from the portable electronic device, asensory-feedback instruction to the portable electronic device that,when performed by the portable electronic device, indicates that thechange in the state of the entertainment device is being implemented.